Computers and information technology may be quite complex. From the software and middleware to the hardware and connections for communicating between machines, the technologies such as hardware, protocols, languages, software packages, and software standards involved is large and changes rapidly. Accordingly, companies, governments, associations and other organizations rely on proficient programmers, network administrators, database administrators, and other IT (Information Technology) professionals to keep their computer systems functioning properly and efficiently.
To become and remain employable, these IT professionals must be able to keep up and prove they can provide the services the employer seeks, often before being offered an IT employment position. Due to the high costs and high risks of using unskilled employees, employers are often unwilling to allow employees to learn the necessary skills on the job. In addition, universities and colleges do not teach all of the specific programming languages, hardware devices, or protocols used in the industry. Instead, using a select few, they focus on theory and principles common across the technologies.
Thus, to provide the credentials demanded by employers, the IT industry uses certification exams, particularly in the software industry. This effort is often led by vendors of the different technologies. The certification asserts that the vendor considers the certificate holder competent to use and maintain the IT technology being certified. Novell®, Microsoft® and others offer certification exams such as CNE (Certified NetWare Engineer), MCP (Microsoft Certified Professional), MCSE (Microsoft Certified Systems Engineer), and others.
However, conventional certification exams which use multiple choice and fill-in-the-blank questions are limited in their ability to measure behavioral skills. Some examinees may simply be good at memorizing rules and appropriate answers but lack the skills needed to perform “real world” tasks. Alternatively, the examinees may prepare for the exam by storing the answers in short term memory. Once the exam is over and the examinee has passed, the facts are quickly forgotten.
Essentially, the examinee lacks the kind of knowledge which is best obtained through the experience of applying the concepts and principles in real life situations or in a simulated environment. As competition has increased in the IT field, the certification exams have begun to focus greater proportions of their exams on testing an examinee's knowledge and skill in relation to an IT system rather than their ability to retain and recall technical information. This means the exams are setting objectives for the examinees which conventional test questions fail to adequately meet or measure.
For example, rather than require that an examinee know how to configure a network card. The certification exams are requiring that the examinee demonstrate an ability to configure the network card. This kind of objective is best assessed using a simulation or an actual IT environment.
The certification exams of today are designed to test an examinee's competence in using the particular technology, software, administration tool, or the like. These are the skills the employers want to know an applicant possesses. Crafting an exam question to test an examinee's ability to use the technology is very difficult without conducting a one-on-one assessment much like an actual driving test with a DMV official to receive a driver's license. Generally, due to the number of exams given and the number of examinees, one-on-one assessments are impractical.
To address these limitations, interactive exam questions have been developed which are directed toward an examinee's behavior. The examinee must do something to demonstrate that they could perform a specific task in an actual system, if required to do so. These interactive exam questions provide some realism when compared against standard exam questions. For example, one or more screen shots of a tool, configuration window, or other administrative context may be displayed to provide context for the question. The user may then be asked to identify which parameter is set incorrectly. The user may identify the parameter by dragging and dropping a correct parameter setting to an answer box.
Alternatively, an image of the software window or tool of interest may be displayed and the user may be asked to point to the configuration setting which should be changed to accomplish a desired result. These types of interactive exam questions are referred to as “point-and-shoot” questions because the examinee uses a mouse or other pointer to indicate the field or other user interface control which must be modified to properly answer the question.
Alternatively, a conventional simulation may be used. Conventional simulations include a multi-media presentation of images and user interface controls which allow very limited interaction between the user and the simulation. These simulations may provide an interface which allows an examinee to navigate through a simulated software product to the proper location to resolve a problem described in an exam question. However, these simulations only provide user interface functionality for the menus, windows, and controls necessary to navigate to the proper software window to answer the question and to no other parts of the environment. Because the other menus, controls, and windows do not function, an examinee can find the proper screen or problem context quickly by simply using trial and error, at which point answering the question or performing a task may be trivial. Simulations executed by such simulations are often referred to as “closed” or “dumb” simulations because of the limited functionality and navigation paths.
In contrast, an “open” or “intelligent” simulation is one which realistically simulates a computer environment within which a simulation scenario takes place. The open simulation includes logic, rules, and processing ability which allows changes made by a user in the simulation to have a corresponding effect in the simulation. In addition, the open simulation matches the behavior of the actual components of the computer environment so that a user may take multiple navigation paths. Providing multiple navigation paths and ways to perform a task allows a user to truly “trouble shoot” a problem as will be expected in the actual computer environment. The user is free to investigate the simulated computer environment to solve a problem in a manner similar to real life.
Unfortunately, conventional simulators which provide limited openness in the simulations have other limitations. Generally, the data defining a simulation and/or computer environment is tightly integrated with the executable code of the simulators. Therefore, development and revisions to simulations is very expensive because this must be done by programmers instead of less expensive test item writers. The programmers modify the actual simulators to produce different simulations.
In addition, developing truly “open” simulators which simulate each feature, function, and aspect of an actual system is very time consuming, expensive, and complex. To shorten the development cycle and hold down costs, only certain features and functions of the actual system are simulated. Consequently, conventional simulations which are available provide only limited “openness”.
These conventional simulations are often developed using multi-media development tools such as Shockwave or Flash products from Macromedia, of San Francisco, Calif. Generally, these tools provide limited flexibility. Each Shockwave or Flash application is programmed for a specific simulation exam question. The conventional simulation created is proprietary, customized, and specific. The training required to use these multi-media development tools generally limits their use to computer programmers. Any subsequent modifications to a simulation is done by the programmer. Using a programmer to develop or modify a conventional simulation increases simulation production costs.
Furthermore, these conventional simulations generally do not provide functionality for recording an examinee's response and/or interactions with the program. Conventional simulations are designed for presenting information, not monitoring an examinee's responses. Determining whether the examinee correctly resolved a problem presented is often rudimentary or non-existent. Typically, conventional simulations are equipped to provide a binary type of a score. The simulation indicates whether or not the user completed the entire simulation correctly or not.
Since simulations in certification exams are still relatively new, the industry has focused efforts on improving the simulations, and the simulators which present them, instead of evaluation technology for the simulations. However, as simulations become more mainstream, the industry's focus will likely shift to the simulation evaluation process to assist in distinguishing among the most qualified users who have ‘passed’ the exam. A simple passing score may no longer be sufficient.
Accordingly, what is needed is a system and method that overcomes the problems and disadvantages of the prior art. The system and method should simulate a computer environment, such that a user is provided with a realistic experience in configuring systems, trouble shooting and resolving problems. The system and method should provide a simulated computer environment which allows an author of a simulation to automatically define a set of possible tasks for the simulation by performing the tasks in the simulated computer environment. The system and method should provide a robust evaluation process for simulations. The system and method should provide an evaluation process which allows for partial scores and/or weighted scores to be awarded for performance in a simulation, as well as conventional binary scores. The system and method should provide an evaluation process which allows for re-scoring and simulation exam question seeding to improve the quality of the simulations developed.